Decode audio in the Blocks API, as RawAudioSamples - #1666
Merged
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1666
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
NicolasHug
force-pushed
the
audio-blocks-decoder
branch
2 times, most recently
from
August 21, 2026 15:58
4c36097 to
3684786
Compare
NicolasHug
force-pushed
the
audio-blocks-decoder
branch
from
August 21, 2026 15:58
3684786 to
2eccb52
Compare
PacketDecoder now follows the demuxer it was built from: RawFrames for a VideoDemuxer, RawAudioSamples for an AudioDemuxer. Decoding is the same avcodec_send_packet / avcodec_receive_frame pair either way, so this stays one block rather than two. RawAudioSamples carries the samples in the codec's own sample type (uint8 for u8, int16 for s16, float32 for flt...), always as a contiguous [num_channels, num_samples] tensor. That is a copy, not a view: planar formats put each channel in its own allocation and packed ones interleave them, so neither is that shape as it stands, and an audio frame is only a few kB. Normalizing to [-1, 1] is the converter's job, not the decoder's. Audio is CPU-only, so a non-CPU device raises rather than being silently ignored, and an unspecified one doesn't inherit a non-CPU default device.
NicolasHug
force-pushed
the
audio-blocks-decoder
branch
from
August 21, 2026 15:58
2eccb52 to
4925a6a
Compare
The blocks decoder now handles audio, producing RawAudioSamples. Decoding is the same avcodec_send_packet / avcodec_receive_frame pair for both media, so the C++ PacketDecoder stays one class; the split is Python-only, so that each decoder has an exact output type and only the arguments that apply to it. AudioPacketDecoder has no device parameter at all: audio is CPU-only, and that shouldn't change with torch.set_default_device. RawAudioSamples carries the samples in the codec's own sample type (uint8 for u8, int16 for s16, float32 for flt...), always as a contiguous [num_channels, num_samples] tensor. That is a copy, not a view: planar formats put each channel in its own allocation and packed ones interleave them, so neither is that shape as it stands, and an audio frame is only a few kB. Normalizing to [-1, 1] is the converter's job, not the decoder's. is_mpeg_ps_ now applies to audio too. Seeking an MPEG program stream lands on a container-level byte offset, so the parser resumes mid-frame and rebuilds invalid packets until it resyncs - a property of the container, not of the codec. Gating it to video made seeking sine_stereo_mp2.mpg's audio stream fail outright with AVERROR_INVALIDDATA; there's now a test for it.
NicolasHug
force-pushed
the
audio-blocks-decoder
branch
from
August 21, 2026 16:45
900f328 to
15311a0
Compare
NicolasHug
added a commit
that referenced
this pull request
Aug 24, 2026
- get_swr_output_num_samples_bound is now declared and defined by the parent commit, so remove this branch's copy. - swr_convert_to_tensor keeps the parent's `const uint8_t**` parameter, which is what compiles against FFmpeg 4-6 as well as 7+. - PacketDecoder became AudioPacketDecoder in #1666; update the converter tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PacketDecoder now follows the demuxer it was built from: RawFrames for a
VideoDemuxer, RawAudioSamples for an AudioDemuxer. Decoding is the same
avcodec_send_packet / avcodec_receive_frame pair either way, so this stays
one block rather than two.
RawAudioSamples carries the samples in the codec's own sample type (uint8
for u8, int16 for s16, float32 for flt...), always as a contiguous
[num_channels, num_samples] tensor. That is a copy, not a view: planar
formats put each channel in its own allocation and packed ones interleave
them, so neither is that shape as it stands, and an audio frame is only a
few kB. Normalizing to [-1, 1] is the converter's job, not the decoder's.
Audio is CPU-only, so a non-CPU device raises rather than being silently
ignored, and an unspecified one doesn't inherit a non-CPU default device.
Stack created with GitHub Stacks CLI • Give Feedback 💬